Network For Emotion Recognition Research Articles

A Three-stage multimodal emotion recognition network based on text low-rank fusion

A Three-stage multimodal emotion recognition network based on text low-rank fusion

Detecting emotions through EEG signals based on modified convolutional fuzzy neural network

Emotion is a human sense that can influence an individual’s life quality in both positive and negative ways. The ability to distinguish different types of emotion can lead researchers to estimate the current situation of patients or the probability of future disease. Recognizing emotions from images have problems concealing their feeling by modifying their facial expressions. This led researchers to consider Electroencephalography (EEG) signals for more accurate emotion detection. However, the complexity of EEG recordings and data analysis using conventional machine learning algorithms caused inconsistent emotion recognition. Therefore, utilizing hybrid deep learning models and other techniques has become common due to their ability to analyze complicated data and achieve higher performance by integrating diverse features of the models. However, researchers prioritize models with fewer parameters to achieve the highest average accuracy. This study improves the Convolutional Fuzzy Neural Network (CFNN) for emotion recognition using EEG signals to achieve a reliable detection system. Initially, the pre-processing and feature extraction phases are implemented to obtain noiseless and informative data. Then, the CFNN with modified architecture is trained to classify emotions. Several parametric and comparative experiments are performed. The proposed model achieved reliable performance for emotion recognition with an average accuracy of 98.21% and 98.08% for valence (pleasantness) and arousal (intensity), respectively, and outperformed state-of-the-art methods.

EEG-ConvNet: Convolutional networks for EEG-based subject-dependent emotion recognition

Biomedical researchers face a significant challenge in identifying emotions from electroencephalogram (EEG) signals due to their intricate and dynamic nature. The deep learning (DL) models, particularly the convolutional neural networks (CNNs), have shown significant potential in identifying the emotions for the EEG signals. However, most current DL models require complex feature engineering implicated in increased computational complexities. This research introduces a new CNN, i.e., the EEG-ConvNet, to overcome these limitations and challenges. The proposed EEG-ConvNet comprises five convolutional layers with batch normalization and max pooling. In addition, fine-tuning techniques improve the validation of pre-trained models. The study also employs the Short-time Fourier transform (STFT) and Mel spectrograms involving EEG signals from the SEED dataset. The suggested approaches effectively extract and organize emotion-related information from simple 2D spectrograms derived from 1D EEG data. The pre-trained GoogLeNet and ResNet-34 models are fine-tuned on these simple spectrograms to discover relevant features. For interpretability, the study employs explainable artificial intelligence (XAI) methods, specifically Gradient class activation mapping (Grad-CAM) and integrated gradients (IG). The STFT-based GoogLeNet and ResNet-34 models achieve accuracies of 99.97% and 99.95%, respectively. The Mel spectrogram-based GoogLeNet and ResNet-34 models achieve accuracies of 99.49% and 99.31%, respectively. The suggested EEG-ConvNet achieves an accuracy of 99.03% on STFT spectrograms. The EEG-ConvNet has a prediction time of only 6.5 ms, paving the way for real-time emotion recognition. While comparing with the previously published DL models, the proposed classification models exhibit better classification performances on the common SEED dataset.

AC-CfC: An attention-based convolutional closed-form continuous-time neural network for raw multi-channel EEG-based emotion recognition

Emotion recognition based on electroencephalogram (EEG) is a critical task in the field of affective brain-computer interfaces. However, due to the non-stationarity and individual variability of EEG, hand-designed features cannot adequately capture the nonlinear and high-dimensional properties of raw EEG. Currently, spatial–temporal models have been verified to effectively capture the spatial–temporal information for EEG. However, these models are characterized by complex structures, large parameter counts, and the need for extensive training data. To overcome these disadvantages, this paper propose an end-to-end spatial–temporal model for emotion recognition based on raw EEG, called attention-based convolutional closed-form continuous-time neural network (AC-CfC). This model employs a channel attention mechanism to weight and encode EEG, capturing channel dependencies. Subsequently, one-dimensional convolutional neural networks and closed-form continuous-time neural networks are used to extract deep spatial–temporal features. Additionally, an adaptive loss-controlling mechanism is designed to enhance the model's decision-making ability between classes that are easily confused. To verify the effectiveness of the proposed model, experiments are conducted on the DEAP and DREAMER datasets. The average accuracies of the proposed model reach to 94.76% and 93.01% for valence and arousal in subject-independent experiments on the DEAP dataset, with an improvement of 11.8% and 8.73% respectively compared with the state-of-art model ACRNN. On the DREAMER dataset, the average accuracies reach to 81.83%, 81.22%, and 80.63% for valence, arousal and dominance, with an improvement of 2.55%, 6.64%, and 6.98% respectively over ACRNN. These results show that our proposed model exhibits better performance than some state-of-art models in the same category.

Newman-Watts-Strogatz topology in deep echo state networks for speech emotion recognition

Cortical neural connectivity exhibits a small-world (SW) network topology. However, its role in neural information processing remains unclear. In this study, we investigate the learning performance of a Deep Echo State Network (DeepESN) that incorporates the Newman-Watts-Strogatz (NWS) graph topology as a reservoir. The short path length of the SW topology allows input information to reach output nodes faster, and the clustered arrangement of this topology can expand the range of the echo state property. Additionally, in this paper, we seek to identify the best set of these key factors, such as the spectral radius of the reservoir’s weight matrix and the hyperparameters neighbors' and probabilities. We evaluate the NWS-DeepESNs on four Speech Emotion Recognition (SER) tasks, which are RAVDESS, Emo-db, SAVEE, and TESS databases. By examining the NWS-DeepESN’s performance on SER tasks, we aim to determine the role of the NWS graph topology in enhancing neural information processing. The obtained results exhibited significant improvements in recognition accuracies (weighted and unweighted) with (76.45%, 75.79%), (87.89%, 87.14%), (99%, 98.62%), and (63%, 62.24%), respectively, for RAVDESS, Emo-db, SAVEE, and TESS databases. The proposed approach improves emotion recognition capabilities and yields more accurate results compared to existing state-of-the-art approaches.

Exploring non-Euclidean relationships in EEG emotion recognition: A dynamic graph attention network

Deep learning classification models based on electroencephalogram (EEG) emotion recognition have demonstrated considerable proficiency in the categorization of emotional states. However, these models have limitations in their capability to analyze the active states and cooperative relationships among distinct brain regions. This study proposes a dynamic graph attention network (DGAT) for EEG emotion recognition, which learns the features of each channel and leverages multiple-head self-attention mechanisms to capture non-Euclidean relationships between channels. Then, we use differential entropy features of emotions signals on the SJTU emotion EEG dataset (SEED). The DGAT model achieved improved subject-dependent and cross-subject classification accuracy compared to previous models. Moreover, ablation studies show that the channel weight matrix(CWM) and appropriate hyper-parameters can improve the performance of the DGAT model significantly. Furthermore, by conducting interpretable analysis of the new connections and electrode weights learned by the model, we find that these connection weight relationships reflect a certain degree of coordination within the brain for EEG-based emotion recognition. These findings provide a new method for EEG emotion recognition and highlights the potential for using deep learning models to analyze the active state and synergistic relationships among brain regions.

Multiple-source distribution deep adaptive feature norm network for EEG emotion recognition

Multiple-source distribution deep adaptive feature norm network for EEG emotion recognition

Concept-guided multi-level attention network for image emotion recognition

Concept-guided multi-level attention network for image emotion recognition

AN OPTIMIZED RESIDUAL NETWORK FOR EMOTION RECOGNITION BASED ON A MULTI-FEATURES FUSION TECHNOLOGY AND ELECTROENCEPHALOGRAPHY SIGNALS

Presently, a significant number of students enrolled in colleges and universities encounter mental health challenges, including academic stress and difficulties in interpersonal relationships. These factors contribute to a decline in mental well-being and necessitate prompt intervention and assistance. The automation of mental health identification in students greatly benefits from the implementation of intelligent emotion recognition systems. Through the analysis of students’ emotional states, educational institutions can gain a deeper understanding of students’ mental well-being, enabling them to promptly identify and address any issues and offer more complete care and support to students. This paper proposes a method for automated emotion recognition that is driven by data. The method is developed based on the provided background information. The proposed approach incorporates a feature fusion mechanism and an attention mechanism into the ResNet-34 model. This integration enhances the model’s capability to analyze intricate details and subsequently improves its classification performance when applied to electroencephalography (EEG) signals. This paper introduces several key innovations. First, an optimization technique is applied to the input component of the model, enabling multifeature fusion. Second, an attention mechanism is incorporated after the residual network module, enabling the model to prioritize parts that contribute to classification and enhance feature extraction. Finally, the network parameters are optimized using both softmax loss and center loss functions. The findings from the analysis of the sentiment EEG public data SJTU Emotion EEG Dataset (SEED) indicate that the proposed sentiment recognition approach not only enhances the classification performance of the model on the sentiment EEG data but also improves the stability of the results. This paper presents a novel approach that enables automatic and efficient recognition of students’ emotions on commonly used platforms. The findings of this study hold significant implications for mental health assessment and detection in real-life production settings, offering substantial reference value in this domain.

Brain-inspired modular echo state network for EEG-based emotion recognition.

Previous studies have successfully applied a lightweight recurrent neural network (RNN) called Echo State Network (ESN) for EEG-based emotion recognition. These studies use intrinsic plasticity (IP) and synaptic plasticity (SP) to tune the hidden reservoir layer of ESN, yet they require extra training procedures and are often computationally complex. Recent neuroscientific research reveals that the brain is modular, consisting of internally dense and externally sparse subnetworks. Furthermore, it has been proved that this modular topology facilitates information processing efficiency in both biological and artificial neural networks (ANNs). Motivated by these findings, we propose Modular Echo State Network (M-ESN), where the hidden layer of ESN is directly initialized to a more efficient modular structure. In this paper, we first describe our novel implementation method, which enables us to find the optimal module numbers, local and global connectivity. Then, the M-ESN is benchmarked on the DEAP dataset. Lastly, we explain why network modularity improves model performance. We demonstrate that modular organization leads to a more diverse distribution of node degrees, which increases network heterogeneity and subsequently improves classification accuracy. On the emotion arousal, valence, and stress/calm classification tasks, our M-ESN outperforms regular ESN by 5.44, 5.90, and 5.42%, respectively, while this difference when comparing with adaptation rules tuned ESNs are 0.77, 5.49, and 0.95%. Notably, our results are obtained using M-ESN with a much smaller reservoir size and simpler training process.

Cerebral asymmetry representation learning-based deep subdomain adaptation network for electroencephalogram-based emotion recognition.

Objective.Extracting discriminative spatial information from multiple electrodes is a crucial and challenging problem for electroencephalogram (EEG)-based emotion recognition. Additionally, the domain shift caused by the individual differences degrades the performance of cross-subject EEG classification.Approach.To deal with the above problems, we propose the cerebral asymmetry representation learning-based deep subdomain adaptation network (CARL-DSAN) to enhance cross-subject EEG-based emotion recognition. Specifically, the CARL module is inspired by the neuroscience findings that asymmetrical activations of the left and right brain hemispheres occur during cognitive and affective processes. In the CARL module, we introduce a novel two-step strategy for extracting discriminative features through intra-hemisphere spatial learning and asymmetry representation learning. Moreover, the transformer encoders within the CARL module can emphasize the contributive electrodes and electrode pairs. Subsequently, the DSAN module, known for its superior performance over global domain adaptation, is adopted to mitigate domain shift and further improve the cross-subject performance by aligning relevant subdomains that share the same class samples.Main Results.To validate the effectiveness of the CARL-DSAN, we conduct subject-independent experiments on the DEAP database, achieving accuracies of 68.67% and 67.11% for arousal and valence classification, respectively, and corresponding accuracies of 67.70% and 67.18% on the MAHNOB-HCI database.Significance.The results demonstrate that CARL-DSAN can achieve an outstanding cross-subject performance in both arousal and valence classification.

MISNet: multi-source information-shared EEG emotion recognition network with two-stream structure.

When constructing machine learning and deep neural networks, the domain shift problem on different subjects complicates the subject independent electroencephalography (EEG) emotion recognition. Most of the existing domain adaptation methods either treat all source domains as equivalent or train source-specific learners directly, misleading the network to acquire unreasonable transfer knowledge and thus resulting in negative transfer. This paper incorporates the individual difference and group commonality of distinct domains and proposes a multi-source information-shared network (MISNet) to enhance the performance of subject independent EEG emotion recognition models. The network stability is enhanced by employing a two-stream training structure with loop iteration strategy to alleviate outlier sources confusing the model. Additionally, we design two auxiliary loss functions for aligning the marginal distributions of domain-specific and domain shared features, and then optimize the convergence process by constraining gradient penalty on these auxiliary loss functions. Furthermore, the pre-training strategy is also proposed to ensure that the initial mapping of shared encoder contains sufficient emotional information. We evaluate the proposed MISNet to ascertain the impact of several hyper-parameters on the domain adaptation capability of network. The ablation experiments are conducted on two publically accessible datasets SEED and SEED-IV to assess the effectiveness of each loss function. The experimental results demonstrate that by disentangling private and shared emotional characteristics from differential entropy features of EEG signals, the proposed MISNet can gain robust subject independent performance and strong domain adaptability.

Hypergraph Neural Network for Emotion Recognition in Conversations

Modeling conversational context is an essential step for emotion recognition in conversations. Existing works still suffer from insufficient utilization of local context information and remote context information. This article designs a hypergraph neural network, namely HNN-ERC, to better utilize local and remote contextual information. HNN-ERC combines the recurrent neural network with the conventional hypergraph neural network to strengthen connections between utterances and make each utterance receive information from other utterances better. The proposed model has empirically achieved state-of-the-art results on three benchmark datasets, demonstrating the effectiveness and superiority of the new model.

ST-SCGNN: A Spatio-Temporal Self-Constructing Graph Neural Network for Cross-Subject EEG-Based Emotion Recognition and Consciousness Detection.

In this paper, a novel spatio-temporal self-constructing graph neural network (ST-SCGNN) is proposed for cross-subject emotion recognition and consciousness detection. For spatio-temporal feature generation, activation and connection pattern features are first extracted and then combined to leverage their complementary emotion-related information. Next, a self-constructing graph neural network with a spatio-temporal model is presented. Specifically, the graph structure of the neural network is dynamically updated by the self-constructing module of the input signal. Experiments based on the SEED and SEED-IV datasets showed that the model achieved average accuracies of 85.90% and 76.37%, respectively. Both values exceed the state-of-the-art metrics with the same protocol. In clinical besides, patients with disorders of consciousness (DOC) suffer severe brain injuries, and sufficient training data for EEG-based emotion recognition cannot be collected. Our proposed ST-SCGNN method for cross-subject emotion recognition was first attempted in training in ten healthy subjects and testing in eight patients with DOC. We found that two patients obtained accuracies significantly higher than chance level and showed similar neural patterns with healthy subjects. Covert consciousness and emotion-related abilities were thus demonstrated in these two patients. Our proposed ST-SCGNN for cross-subject emotion recognition could be a promising tool for consciousness detection in DOC patients.

ERNetCL: A novel emotion recognition network in textual conversation based on curriculum learning strategy

Emotion recognition in conversation (ERC) has emerged as a research hotspot in domains such as conversational robots and question-answer systems. How to efficiently and adequately retrieve contextual emotional cues has been one of the key challenges in the ERC task. Existing efforts do not fully model the context and employ complex network structures, resulting in limited performance gains. In this paper, we propose a novel emotion recognition network based on curriculum learning strategy (ERNetCL). The proposed ERNetCL primarily consists of temporal encoder (TE), spatial encoder (SE), and curriculum learning (CL) loss. We utilize TE and SE to combine the strengths of previous methods in a simplistic manner to efficiently capture temporal and spatial contextual information in the conversation. To ease the harmful influence resulting from emotion shift and simulate the way humans learn curriculum from easy to hard, we apply the idea of CL to the ERC task to progressively optimize the network parameters. At the beginning of training, we assign lower learning weights to difficult samples. As the epoch increases, the learning weights for these samples are gradually raised. Extensive experiments on four datasets exhibit that our proposed method is effective and dramatically beats other baseline models.

Attention-based 3D convolutional recurrent neural network model for multimodal emotion recognition.

Multimodal emotion recognition has become a hot topic in human-computer interaction and intelligent healthcare fields. However, combining information from different human different modalities for emotion computation is still challenging. In this paper, we propose a three-dimensional convolutional recurrent neural network model (referred to as 3FACRNN network) based on multimodal fusion and attention mechanism. The 3FACRNN network model consists of a visual network and an EEG network. The visual network is composed of a cascaded convolutional neural network-time convolutional network (CNN-TCN). In the EEG network, the 3D feature building module was added to integrate band information, spatial information and temporal information of the EEG signal, and the band attention and self-attention modules were added to the convolutional recurrent neural network (CRNN). The former explores the effect of different frequency bands on network recognition performance, while the latter is to obtain the intrinsic similarity of different EEG samples. To investigate the effect of different frequency bands on the experiment, we obtained the average attention mask for all subjects in different frequency bands. The distribution of the attention masks across the different frequency bands suggests that signals more relevant to human emotions may be active in the high frequency bands γ (31-50 Hz). Finally, we try to use the multi-task loss function Lc to force the approximation of the intermediate feature vectors of the visual and EEG modalities, with the aim of using the knowledge of the visual modalities to improve the performance of the EEG network model. The mean recognition accuracy and standard deviation of the proposed method on the two multimodal sentiment datasets DEAP and MAHNOB-HCI (arousal, valence) were 96.75 ± 1.75, 96.86 ± 1.33; 97.55 ± 1.51, 98.37 ± 1.07, better than those of the state-of-the-art multimodal recognition approaches. The experimental results show that starting from the multimodal information, the facial video frames and electroencephalogram (EEG) signals of the subjects are used as inputs to the emotion recognition network, which can enhance the stability of the emotion network and improve the recognition accuracy of the emotion network. In addition, in future work, we will try to utilize sparse matrix methods and deep convolutional networks to improve the performance of multimodal emotion networks.

Layer-Adapted Implicit Distribution Alignment Networks for Cross-Corpus Speech Emotion Recognition

Layer-Adapted Implicit Distribution Alignment Networks for Cross-Corpus Speech Emotion Recognition

Attention-based Temporal Graph Representation Learning for EEG-based Emotion Recognition.

Due to the objectivity of emotional expression in the central nervous system, EEG-based emotion recognition can effectively reflect humans' internal emotional states. In recent years, convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have made significant strides in extracting local features and temporal dependencies from EEG signals. However, CNNs ignore spatial distribution information from EEG electrodes; moreover, RNNs may encounter issues such as exploding/vanishing gradients and high time consumption. To address these limitations, we propose an attention-based temporal graph representation network (ATGRNet) for EEG-based emotion recognition. Firstly, a hierarchical attention mechanism is introduced to integrate feature representations from both frequency bands and channels ordered by priority in EEG signals. Second, a graph convolutional neural network with top-k operation is utilized to capture internal relationships between EEG electrodes under different emotion patterns. Next, a residual-based graph readout mechanism is applied to accumulate the EEG feature node-level representations into graph-level representations. Finally, the obtained graph-level representations are fed into a temporal convolutional network (TCN) to extract the temporal dependencies between EEG frames. We evaluated our proposed ATGRNet on the SEED, DEAP and FACED datasets. The experimental findings show that the proposed ATGRNet surpasses the state-of-the-art graph-based mehtods for EEG-based emotion recognition.

A Multi-Level Alignment and Cross-Modal Unified Semantic Graph Refinement Network for Conversational Emotion Recognition

A Multi-Level Alignment and Cross-Modal Unified Semantic Graph Refinement Network for Conversational Emotion Recognition

A Graph Neural Network for EEG-Based Emotion Recognition With Contrastive Learning and Generative Adversarial Neural Network Data Augmentation

A Graph Neural Network for EEG-Based Emotion Recognition With Contrastive Learning and Generative Adversarial Neural Network Data Augmentation